Skip to content

Use provider C# reference map with pre-write internalization#10976

Open
live1206 wants to merge 9 commits into
microsoft:mainfrom
live1206:mtg-hybrid-reference-map
Open

Use provider C# reference map with pre-write internalization#10976
live1206 wants to merge 9 commits into
microsoft:mainfrom
live1206:mtg-hybrid-reference-map

Conversation

@live1206

@live1206 live1206 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the C# generator's Roslyn-generated reference-map/internalization path with a provider-based reference map that runs before files are written. The generator now computes reachability and accessibility from the in-memory provider model, filters unreachable providers up front, and removes stale generated files by cleaning src/Generated before writing.

What changed

  • Build generated-code reachability from provider metadata, signatures, expressions/bodies, helper dependencies, custom-code roots, API-baseline roots, and provider relationships instead of Roslyn-generated documents.
  • Apply internal/public accessibility before output by updating providers directly, including model-factory filtering and XML-doc preservation for internalized types.
  • Remove the generated-code Roslyn removal/internalization post-processing path and related stale-file preservation logic.
  • Avoid stale generated-source state affecting current output, including old generated accessibility roots and stale ModelFactory usings.
  • Keep generated helper, serialization, MRW, custom-code body, and request-body dependencies tied to actual reachable generated code, including parent-namespace custom roots and provider/custom MRW body dependencies.
  • Avoid plugin-build stdout/stderr deadlocks by draining both streams asynchronously.

Performance

Latest full-generation benchmark on the updated benchmark branch (mtg-manual-name-reduction-latest-bench, commit 4cf86686e) averaged 544.5 ms / 53.85 MB with the provider reference map enabled vs 847.4 ms / 138.87 MB with it disabled: 35.7% faster and 61.2% less allocation.

Run Provider reference map off Provider reference map on Speedup Allocation off Allocation on Allocation reduction
1 822.2 ms 531.6 ms 35.3% 138.35 MB 51.64 MB 62.7%
2 891.8 ms 567.5 ms 36.4% 138.16 MB 53.88 MB 61.0%
3 828.1 ms 534.4 ms 35.5% 140.10 MB 56.03 MB 60.0%
Average 847.4 ms 544.5 ms 35.7% 138.87 MB 53.85 MB 61.2%

Benchmark artifacts/logs were saved locally under /tmp/typespec-benchmark-results-latest-runs-20260710030912.

Azure SDK management-plane local project-reference regen speedups:

Latest measured PR head: 434e76802 (test(csharp): rename body dependency reference map tests). Each package was regenerated one-by-one with RegenSdkLocal.ps1 -Parallel 1 using the local Azure SDK for .NET project-reference setup.

SDK package Main baseline This PR Improvement
Azure.ResourceManager.Network 00:37:01.9 00:07:00.9 81.1% faster / 5.28x
Azure.ResourceManager.DataFactory 00:13:43.4 00:03:26.2 75.0% faster / 3.99x
Azure.ResourceManager.AppService 00:10:46.9 00:03:17.6 69.5% faster / 3.27x

Validation

  • Latest targeted local Azure management-plane regens for Azure.ResourceManager.Network, Azure.ResourceManager.DataFactory, and Azure.ResourceManager.AppService passed one-by-one with no public API listing diffs.
  • C# emitter build and focused generator/reference-map tests pass.
  • Full Azure data-plane regen completed with no tracked SDK/library diffs.
  • Targeted data-plane regen/build checks pass with zero targeted generated diffs for:
    • Azure.Data.AppConfiguration (ResponseError MRW context coverage)
    • Azure.AI.Projects (custom-code parent-namespace body dependency root coverage)
    • Azure.AI.Extensions.OpenAI (polymorphic/unknown model retention coverage)
  • Latest main has been merged into the PR branch.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp label Jun 12, 2026
@pkg-pr-new

This comment was marked as outdated.

@github-actions

This comment was marked as off-topic.

@azure-sdk-automation

azure-sdk-automation Bot commented Jun 19, 2026

Copy link
Copy Markdown

You can try these changes here

🛝 Playground 🌐 Website 🛝 VSCode Extension

@live1206 live1206 marked this pull request as ready for review June 25, 2026 08:58
@live1206 live1206 changed the title [Draft] Use hybrid C# reference map for post-processing Use hybrid C# reference map for post-processing Jun 25, 2026
@live1206 live1206 changed the title Use hybrid C# reference map for post-processing Use hybrid C# reference map with pre-write internalization Jun 29, 2026
@live1206 live1206 changed the title Use hybrid C# reference map with pre-write internalization Use provider C# reference map with pre-write internalization Jun 29, 2026
JoshLove-msft

This comment was marked as outdated.

JoshLove-msft

This comment was marked as outdated.

@JoshLove-msft

Copy link
Copy Markdown
Contributor

Correction to my follow-up review (point #1). @live1206 rightly pointed out that RequestHeaderExtensions is a TypeProviderPipelineRequestHeadersExtensionsDefinition (ScmOutputLibrary.cs:90) — not a raw static file. My "there's no provider to mark" reasoning was wrong; please disregard it.

The real reason the textual .SetDelimited( scan / IsRequestHeadersExtensionsFile special-case is needed is different: the only reference to this provider is the request.Headers.SetDelimited(...) call emitted inside RestClientProvider method bodies, and the provider reference graph builds edges from provider metadata (signatures, properties, fields, attributes, declared helper/body deps) — not from arbitrary method-body expressions. RestClientProvider doesn't declare PipelineRequestHeadersExtensions via BuildHelperDependencyNames / BuildBodyDependencyTypes / IncludeGeneratedBodyReferences, so the graph can't see the body-level usage and the file falls back to the string scan.

So the same marker mechanism you introduced for ClientUriBuilder (IncludeGeneratedBodyReferences) could likely subsume this last special-case: have RestClientProvider declare PipelineRequestHeadersExtensions as a helper/body dependency — ideally conditionally, only when a create-request method actually emits SetDelimited — so the graph roots it precisely and the .SetDelimited( scan + HasCustomRequestHeaderExtensionsReference handling can be dropped from the MTG core. Not blocking, just noting it's the same class of body-reference gap the rest of the PR already handles via markers, so it'd be nice to make it consistent rather than a textual exception.

--generated by Copilot

@JoshLove-msft

Copy link
Copy Markdown
Contributor

Design suggestion: derive body references from expression trees rather than declared deps + source re-parsing

Following up on the body-reference discussion — the current approach builds graph edges from provider metadata (signatures, properties, fields, attributes, etc.) and then compensates for body-level references three ways: manually-declared HelperDependencyNames / BodyDependencyTypes / IncludeGeneratedBodyReferences, AddGeneratedBodyReferences re-parsing generated source with Roslyn for a curated set of "candidate" providers, and hardcoded fallbacks (the SetDelimited scan, ChangeTracking helper roots, etc.).

The concern is soundness: a pure-metadata graph is an under-approximation of reachability, because a C# type reference can appear anywhere a method body can (new Helper(), SomeStatic.Call(), extension calls like request.Headers.SetDelimited(...), casts, typeof). For a removal/internalization pass, under-approximating is the dangerous direction — you conclude "nothing references X," internalize/remove it, and the break surfaces later as a downstream compile error or a silent public-API regression. Correctness then rests on provider authors remembering to declare every body dependency (a manually-maintained parallel list that will drift) plus the hand-curated candidate list.

Suggestion: walk the body expression trees instead. The bodies already exist as structured MethodBodyStatement / ValueExpression trees before they're rendered to source. Traversing those to collect CSharpType references (constructor targets, static member owners, type literals, generic args, cast targets) would be:

  • sound by construction for anything the generator itself emits — it closes the gap generically instead of per-provider declaration, and would subsume the SetDelimited special-case and most BodyDependencyTypes/HelperDependencyNames overrides;
  • likely cheaper than AddGeneratedBodyReferences re-parsing generated C# through a Roslyn semantic model.

The fact that the PR re-parses generated source for the client/serialization candidates (rather than walking expression trees) suggests the tree walk wasn't feasible for all bodies — presumably where bodies are raw TypeProvider source or interpolated string literals whose contents aren't structured ValueExpressions. If so, that's the honest residual limitation to document: expression-tree walking would handle the structured majority soundly, with a narrow, explicit fallback (or a CI superset-check against the existing full Roslyn map) for the unstructured remainder — instead of today's broader reliance on manual declarations + name-based special-cases.

Not a merge blocker, but I think it's the change that would turn this from "correct-by-testing" into "correct-by-construction." Would also be worth attributing how much of the ~21% comes from skipping body analysis vs. from pre-write internalization (skipping the Roslyn InternalizeAsync/reduce passes) — if it's mostly the latter, a sound full-body map may retain most of the win.

--generated by Copilot

@JoshLove-msft JoshLove-msft left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found several correctness gaps in the provider-based reference map that can cause generated compilation failures, API compatibility regressions, or missing MRW registrations.

Replace Roslyn-based generated-source reference analysis with provider metadata while preserving generated contracts and customization dependencies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
@live1206 live1206 force-pushed the mtg-hybrid-reference-map branch from 30cc73c to 980c6ad Compare July 14, 2026 01:38
live1206 added 4 commits July 14, 2026 02:45
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
@live1206

Copy link
Copy Markdown
Contributor Author

latest regen: Azure/azure-sdk-for-net#60965 (comment)

live1206 added 2 commits July 15, 2026 01:25
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17

@JoshLove-msft JoshLove-msft left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final pass found one remaining correctness issue in the shared ClientOptions accessibility fix.

--generated by Copilot

@JoshLove-msft

Copy link
Copy Markdown
Contributor

Reran the PR's full-generation benchmark three times at the documented benchmark revision 4cf86686e.

Environment: Windows 11, .NET 10.0.10, 13th Gen Intel Core i7-13700T. The benchmark fixture lookup required an uncommitted Windows-only path-separator normalization; the measured workload and off/on implementations were unchanged.

Run Reference map off Reference map on Speedup Allocation off Allocation on Allocation reduction
1 1,145.5 ms 736.7 ms 35.7% 162.10 MB 70.41 MB 56.6%
2 1,658.7 ms 653.0 ms 60.6% 162.33 MB 72.07 MB 55.6%
3 1,192.2 ms 736.0 ms 38.3% 152.34 MB 69.61 MB 54.3%
Average 1,332.1 ms 708.6 ms 46.8% 158.92 MB 70.70 MB 55.5%

Run 2's off measurement is visibly noisy, but every pass confirms a substantial time improvement and the allocation reduction is stable at 54-57%.

--generated by Copilot

live1206 added 2 commits July 15, 2026 03:57
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4b68173-2f2b-467d-9a94-9be9d14b5e17

protected override IReadOnlyList<MethodBodyStatement> BuildAttributesForWrite()
{
var visitorAttributes = base.BuildAttributesForWrite().Where(static attribute => !IsBuildableAttribute(attribute));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this a special case? Technically, any of the the type providers can be visited so I'm curious why we need to have this special handling here.

var isResolvable = IsResolvableBuildableType(provider.Type);
var implementsMrw = ImplementsModelReaderWriter(provider);
var hasWritableSerialization = HasWritableModelReaderWriterSerialization(provider);
return isResolvable && implementsMrw && hasWritableSerialization;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can improve the perf of this by exiting early on any one of the conditions

public override MethodBodyStatement AddCollectionHeaders(ValueExpression prefix, ValueExpression headers)
=> Original.Property(nameof(PipelineRequest.Headers)).Invoke(nameof(PipelineRequestHeaders.Add), [prefix, headers]).Terminate();
=> Original.Property(nameof(PipelineRequest.Headers))
.Invoke(nameof(PipelineRequestHeaders.Add), [prefix, headers], typeArguments: null, callAsAsync: false, extensionType: new PipelineRequestHeadersExtensionsDefinition().Type)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to use the cached instance of extensionType ?

ValueExpression[] parameters = format != null ? [Literal(name), value, delimiter, format] : [Literal(name), value, delimiter];
return pipelineRequest.Property(nameof(PipelineRequest.Headers)).Invoke("SetDelimited", parameters).Terminate();
return pipelineRequest.Property(nameof(PipelineRequest.Headers))
.Invoke("SetDelimited", parameters, typeArguments: null, callAsAsync: false, extensionType: new PipelineRequestHeadersExtensionsDefinition().Type)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question here

AddMetadataReference(MetadataReference.CreateFromFile(typeof(ClientResult).Assembly.Location));
AddMetadataReference(MetadataReference.CreateFromFile(typeof(BinaryData).Assembly.Location));
AddMetadataReference(MetadataReference.CreateFromFile(typeof(JsonSerializer).Assembly.Location));
AddCustomCodeMethodDependency("SetDelimited", PipelineRequestHeadersExtensionsDefinition);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we make this an internal static property in PipelineRequestHeadersExtensionsDefinition to avoid having to change this in two places

generatedTypeNamespace ?? CustomCodeView?.Type.Namespace ?? BuildNamespace(),
generatedTypeName ?? CustomCodeView?.Name ?? BuildName(),
DeclaringTypeProvider?.Type.Name);
GetDeclaringTypeName(DeclaringTypeProvider));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are doing this calculation in more than one place. Should we consider caching the value ?

}
}

private static bool ShouldKeepGeneratedFile(FileInfo file, HashSet<string> fileNamesToKeep)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think we can simply do away with this helper since it's just delegating to the .Contains method

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants